@LucaCinquini - There is one pending item. Once any DAGs are executed the airflow-celery-workers node remains. I don't know if we want to define this as a core component so that it runs with the other pods or if we want to try setting the consolidationPolicy to WhenEmptyOrUnderutilized so that Karpenter consolidates pods on nodes when the node's resources are underutilized?

@nikki-t : I tested the PR and it looks good - in steady state, the cluster has 3 nodes. 2 would be even better, can you try your suggestion of using "WhenEmptyOrUnderutilized" for the celer-workers, and see if the 3rd node is shut down? Thanks.

@LucaCinquini - I will work on testing this today. For future reference here is the details on consolidation: https://karpenter.sh/v1.0/concepts/disruption/#consolidation

@LucaCinquini - I pushed some changes that seem to have worked. The main thing we will want to decide on is if it is okay to wait for the DAG setup task to run? Because with these changes there is no celery worker node available to run on, so the task has to wait for a node to be launched and initialized.

@nikki-t : I think your last changes are good. To finalize, can you add a comment to the template file that explains what the implications of using 0 vs 1 are? By default, we should set the value to 0.

@LucaCinquini - I added a comment to the template file. Let me know if you want me to add any further details.

Not sure why the pre-commit GitHub action is failing, it seems to not be able to install a dependency.